Information systems generate vast amounts and wide varieties of machine data such as activity logs, configuration files, network messages, database records, etc. This machine data can be useful in troubleshooting systems, detecting operation trends, catching security problems, and measuring business performance. Unfortunately, however, a lack of tools to efficiently process and analyze heterogeneous datasets makes it tedious to mine the machine data for analytic insights. Most machine data such as generated logs, such as syslog and web-server logs, are unstructured text files. While the machine data may have some loosely implied structure, the specific structure varies across systems and environments, and is subject to frequent and unexpected changes. This type of data typically represents a complete history of events over time rather than a snapshot in time and is commonly several orders of magnitude larger than structured datasets.
Due to the large scale and temporal organization of log entries, traditional analysis techniques are often unsuitable for these datasets. Standard relational databases include a set of predefined fields in which machine data may be stored into for later analytics. The data is typically stored in static fields in rows and columns of the database. But once this data is summarized and stored in these predefined database fields, the underlying data is discarded and cannot be later analyzed or used as a basis for new search queries based on different fields or different combinations of fields. Thus despite being goldmines of information, these machine logs as well as other machine data are rarely explored and often deleted to free up storage space.
In addition, conventional approaches do not provide metrics that can be useful in analyzing and processing machine data. First, the relevance of certain events is unclear. For instance, the impact of events from a particular domain to the IT environment may be unclear. This makes it difficult to distinguish high-priority events from lower-priority events. Second, the meaning of machine data may be indirect and unclear. For instance, in the Enterprise Security context, it may be difficult to determine that a large number of access attempts is a problem event because it may be unclear what an access event consists of. Third, in many cases, system metrics are static and cannot be customized to address a particular problem. Finally, in many IT monitoring systems that process machine data, it is difficult to establish what the baseline of the system should be in order to further determine if there is a deviation from that baseline.